Tighten cua-agent design seams: typed translator, declared channels - #21
Merged
Conversation
- The translator now consumes the canonical CuaAction union with an exhaustive switch instead of re-parsing untyped records; malformed shapes can no longer silently coerce to 0,0 clicks, and drift in the canonical vocabulary becomes a compile error. The documented button-coercion contract is the one remaining runtime guard. - cua-ai exports CuaSimpleStreamOptions so the keepToolNames channel the Yutori/Tzafon adapters consume is a declared type instead of a cast. - CuaRuntimeController holds one translator per runtime, shared by the tools and the payload getScreenshot capability. - prepareNextTurn is pass-through (stock pi behavior) until a user hook returns an update or a mid-run model assignment dirties the runtime; the state proxy re-keys off the underlying state object identity. - keys.ts reframed: the alias table absorbs model nondeterminism in key naming generally, not provider-specific spellings. - README: harness quickstart showcases session-backed turns and mid-session provider switching with direct prompt() result handling; computerUseExtra is introduced with its rationale. - docs/architecture.md encodes the boundary: provider differences arrive as CuaRuntimeSpec data, agent returns capabilities via CuaPayloadContext, no provider conditionals in packages/agent. - New tests: a full fake-stream tool-call turn (toolCall -> Kernel batch -> toolResult fed back), prepareNextTurn pass-through, and the one-shot refresh after mid-run model assignment. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
rgarcia
marked this pull request as ready for review
June 12, 2026 01:26
|
Created a monitoring plan for this PR. What this PR does: Refactors the internal mechanics of the Intended effect:
Risks:
Status updates will be posted automatically on this PR as monitoring progresses. |
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Design-review follow-ups for
packages/agent:InternalComputerTranslator.executeBatchnow takes the canonicalCuaAction[]union with an exhaustive switch (neverguard), replacing theRecord<string, unknown>re-parsing layer. pi already validates tool arguments against the typebox schemas beforeexecute, so the defensivetoInt/stringOrfallbacks that could turn malformed input into a click at 0,0 are gone; a new canonical action type is now a compile error here instead of a runtime throw. The one remaining runtime guard is the documented button coercion (wire schemas keepbuttonan open string; out-of-set values coerce to"left"per theCuaMouseButtoncontract). The duplicated button unions are deleted in favor of cua-ai's.keepToolNameschannel: cua-ai exportsCuaSimpleStreamOptions(SimpleStreamOptions+keepToolNames), andCuaAgent's stream wrapper uses it — theas SimpleStreamOptions & {...}cast is gone, so the agent/adapter agreement is compiler-owned.CuaRuntimeControllershares a singleInternalComputerTranslatorbetween the executor tools and the payloadgetScreenshotcapability (previously two instances per runtime).createCuaComputerToolsis unchanged for external composition.prepareNextTurn: the wrapper now returnsundefined(stock pi behavior, no context rebuild) unless a user hook returns an update or a mid-run model assignment marks the runtime dirty — removing the every-turn context reconstruction and its silent-field-drop risk. The state proxy re-keys off the underlying state object's identity so it can't wrap a stale target.CuaAgentHarnessquickstart now showcases the harness (session-backed turns, mid-sessionsetModelprovider switch, directprompt()result handling — it returns the turn's finalAssistantMessage) instead of branch-walking gymnastics;computerUseExtrais introduced with its rationale (not every provider's native vocabulary includes navigation; this is the provider-neutral escape hatch). Snippet typechecks against the built packages.CuaRuntimeSpecdata, the agent supplies capabilities back viaCuaPayloadContext,packages/agent/srccontains no provider names/conditionals (grep-testable), with the deliberate exception for generic model imprecision absorbed by the translator.Test plan
toolCall→ translator → mocked Kernel batch/screenshot →toolResultwith image fed back to the second provider request)prepareNextTurnpass-through when nothing changed; one-shot turn update (new model + refreshed tools) afteragent.state.modelassignment, then pass-through againnpm test -w @onkernel/cua-agent(28 passed, live excluded) and-w @onkernel/cua-ai(88 passed)npm run buildboth workspaces (tsdown) + monorepotsc -bgreene2e.live.test.ts) — recommend a run before the next release since translator internals changed🤖 Generated with Claude Code
Note
Medium Risk
Translator and turn-prep behavior changed in the core browser execution path; malformed actions now error instead of silent mis-clicks, so live provider matrix runs are advisable before release.
Overview
Tightens
@onkernel/cua-agent/@onkernel/cua-aiseams for browser computer-use: provider differences stay data-driven viaCuaRuntimeSpec/CuaPayloadContext, documented indocs/architecture.md.The action translator now takes canonical
CuaAction[]with an exhaustive switch instead of re-parsingRecord<string, unknown>— malformed shapes fail loudly rather than coercing to bad clicks; documented mouse-button coercion to"left"is unchanged.prepareNextTurnreturns stock pi behavior (undefined) unless a user hook updates the turn or a mid-run model change marks the runtime dirty, avoiding per-turn context rebuilds. OneInternalComputerTranslatorper runtime is shared by executor tools and payloadgetScreenshotviabuildCuaComputerTools.CuaAgentpassesCuaSimpleStreamOptions(new in cua-ai 0.3.0) forkeepToolNameswithout a cast. Navigationgotoroutes through typed batch actions; key-alias comments/tests frame generic model key imprecision, not provider-specific logic.Docs/README cover harness session turns, mid-session
setModel, andcomputerUseExtrarationale. Releases: cua-agent 0.3.3, cua-ai 0.3.0.Reviewed by Cursor Bugbot for commit 0d0baf8. Bugbot is set up for automated code reviews on this repo. Configure here.